fix(verifiers): reject non-finite judge scores as invalid output - #495

What and why

A rubric judge's score that is not a finite number counted as a pass. _clamp01 is max(0.0, min(1.0, x)), and min(1.0, nan) returns 1.0, so a NaN criterion scored 1.0. Infinity, 1e999 and quoted strings such as "NaN" or "inf" did the same through float(). Under rubric_binary, aggregate_score turned them into a weighted score of 1.0. An integer too large for a float raised an uncaught OverflowError on the first attempt, with no retry. A judge that writes any of these inflates the eval score.

• _parse_judge_results, which all four rubric formats use, now treats a score that is not a finite number as invalid judge output. The verifier then retries with a correction prompt and, after DEFAULT_MAX_RETRIES, raises as it does for other invalid output.
• The correction prompt and JudgeResponseDiscrepancy name the problem directly ("Each score must be a finite number; criterion id(s) c2 were not"), so a response that was valid JSON is not told it was invalid JSON.
• An integer past Python's digit limit is now an invalid-JSON retry instead of an uncaught ValueError.
• hypothesis joins the dev extra for one property test over parse and aggregate.

Out of scope, and unchanged: a non-numeric score such as "abc" or null still reads as 0.0 in the partial and evidence formats and still crashes rubric_binary aggregation; the trajectory_mistakes parser still catches only JSONDecodeError.

How it was tested

• tst/unit/task_step/verifiers/test_rubrics_verifier_nonfinite_scores.py covers bare tokens, quoted strings and overflowing integers across all four formats, plus the retry and give-up paths on the real RubricsVerifierTaskStep. The first commit adds it as a strict xfail: it xfails on the unfixed code. With the fix, all 54 tests pass, and against the unfixed source all 54 fail.
• The Hypothesis property fails when the fix is reverted, with the first counterexample [(0.0, ('NaN', False))].
• pytest tst/unit/task_step tst/unit/env/task_step: 2420 passed. make lint passes. check_plugin_api.py --base origin/main: no break to the plugin surface.

Confidence Score: 5/5

Summary

The PR rejects non-finite judge scores before rubric scores are built.

• Rubric verifiers reject scores that are not finite numbers.

No actionable issues were found. Tests were inspected but not run during this review.

andrewpark-scaleai explicitly identifies nonnumeric scores and the separate trajectory-mistakes parser as unchanged and out of scope.

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Judge response] --> B[Parse JSON and check criterion ids]
  B --> C{Scores are finite?}
  C -->|Yes| D[Build rubric rows and aggregate]
  C -->|No| E{Attempts remain?}
  E -->|Yes| F[Ask judge to correct scores]
  F --> A
  E -->|No| G[Raise error without saving a score]

_{Reviews (1) · Last reviewed commit: "fix(verifiers): reject non-finite judge ..." · Reviewed by Greptile}